Qwen3-Next-80B-A3B-Instruct is an efficient sparse mixture of experts model developed by the Tongyi Qianwen team. It has a total of 80 billion parameters, with only 3 billion active parameters. The model adopts an innovative hybrid attention mechanism and a MoE architecture with an extremely low activation rate, significantly improving inference efficiency while maintaining strong performance. It natively supports a context length of 262,000 tokens and can be extended to 1 million tokens.

Natural Language ProcessingTransformers

MoE-LLaVA is a large vision-language model based on the Mixture of Experts architecture, achieving efficient multimodal learning through sparse activation parameters

MultimodalTransformers

This research explores the most effective activation functions for sparse large language models, comparing the performance of ReLU, SwiGLU, ReGLU, and squared ReLU in terms of sparse computational efficiency.

Natural Language ProcessingTransformersEnglish